Skip to content

HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP…#4347

Merged
saintstack merged 3 commits intoapache:branch-3.3from
saintstack:branch-3.3.HDFS-16586
May 26, 2022
Merged

HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP…#4347
saintstack merged 3 commits intoapache:branch-3.3from
saintstack:branch-3.3.HDFS-16586

Conversation

@saintstack
Copy link
Contributor

@saintstack saintstack commented May 23, 2022

…… (#4338)

  • HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BPServiceActor IllegalThreadStateException 'fatal exception and exit'

Description of PR

Remove the ThreadGroup used by executor factories; they are unused
and ThreadGroups auto-destroy when their Thread-member count goes to zero.
This behavior is incompatible with the configuration we have on the per-volume
executor which is set to let all threads die if no use inside the
keepalive time.

Signed-off-by: Hexiaoqiao hexiaoqiao@apache.org

Backport to branch-3.3.

How was this patch tested?

By running a downstream test that was failing when this PR was not in place.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@saintstack
HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP… (
45e867e 
apache#4338)

* HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BPServiceActor IllegalThreadStateException 'fatal exception and exit'

Remove the ThreadGroup used by executor factories; they are unused
and ThreadGroups auto-destroy when their Thread-member count goes to zero.
This behavior is incompatible with the configuration we have on the per-volume
executor which is set to let all threads die if no use inside the
keepalive time.

Signed-off-by: Hexiaoqiao <hexiaoqiao@apache.org>
@hadoop-yetus
Copy link

hadoop-yetus commented May 23, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 8s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ branch-3.3 Compile Tests _
+1 💚 mvninstall 39m 18s branch-3.3 passed
+1 💚 compile 1m 26s branch-3.3 passed
+1 💚 checkstyle 1m 6s branch-3.3 passed
+1 💚 mvnsite 1m 36s branch-3.3 passed
+1 💚 javadoc 1m 45s branch-3.3 passed
+1 💚 spotbugs 3m 40s branch-3.3 passed
+1 💚 shadedclient 28m 51s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 19s the patch passed
+1 💚 compile 1m 12s the patch passed
+1 💚 javac 1m 12s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 48s the patch passed
+1 💚 mvnsite 1m 19s the patch passed
+1 💚 javadoc 1m 24s the patch passed
+1 💚 spotbugs 3m 31s the patch passed
+1 💚 shadedclient 28m 11s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 211m 45s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 1m 2s The patch does not generate ASF License warnings.
326m 33s
Reason Tests
Failed junit tests hadoop.hdfs.TestRollingUpgrade
hadoop.hdfs.server.datanode.TestBPOfferService
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4347/1/artifact/out/Dockerfile
GITHUB PR #4347
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux b1d7a7baac52 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 45e867e
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4347/1/testReport/
Max. process+thread count 2521 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4347/1/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@saintstack
Copy link
Contributor Author

saintstack commented May 24, 2022

Running above tests in a loop locally w/ the PR in place pass. Let me push a non-change to get a new test run.

@hadoop-yetus
Copy link

hadoop-yetus commented May 24, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 9s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ branch-3.3 Compile Tests _
+1 💚 mvninstall 41m 21s branch-3.3 passed
+1 💚 compile 1m 31s branch-3.3 passed
+1 💚 checkstyle 1m 4s branch-3.3 passed
+1 💚 mvnsite 1m 35s branch-3.3 passed
+1 💚 javadoc 1m 46s branch-3.3 passed
+1 💚 spotbugs 3m 43s branch-3.3 passed
+1 💚 shadedclient 28m 43s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 20s the patch passed
+1 💚 compile 1m 16s the patch passed
+1 💚 javac 1m 16s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 47s the patch passed
+1 💚 mvnsite 1m 19s the patch passed
+1 💚 javadoc 1m 24s the patch passed
+1 💚 spotbugs 3m 26s the patch passed
+1 💚 shadedclient 28m 11s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 227m 56s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 1m 16s The patch does not generate ASF License warnings.
344m 49s
Reason Tests
Failed junit tests hadoop.hdfs.TestRollingUpgrade
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4347/2/artifact/out/Dockerfile
GITHUB PR #4347
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 5504327bf8bc 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / f3f7401
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4347/2/testReport/
Max. process+thread count 2138 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4347/2/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

hadoop-yetus commented May 24, 2022

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 10s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ branch-3.3 Compile Tests _
+1 💚 mvninstall 37m 59s branch-3.3 passed
+1 💚 compile 1m 25s branch-3.3 passed
+1 💚 checkstyle 1m 7s branch-3.3 passed
+1 💚 mvnsite 1m 36s branch-3.3 passed
+1 💚 javadoc 1m 44s branch-3.3 passed
+1 💚 spotbugs 3m 41s branch-3.3 passed
+1 💚 shadedclient 28m 43s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 19s the patch passed
+1 💚 compile 1m 13s the patch passed
+1 💚 javac 1m 13s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 47s the patch passed
+1 💚 mvnsite 1m 20s the patch passed
+1 💚 javadoc 1m 21s the patch passed
+1 💚 spotbugs 3m 26s the patch passed
+1 💚 shadedclient 28m 8s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 219m 51s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 asflicense 1m 1s The patch does not generate ASF License warnings.
333m 19s
Reason Tests
Failed junit tests hadoop.hdfs.TestRollingUpgrade
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithShortCircuitRead
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4347/3/artifact/out/Dockerfile
GITHUB PR #4347
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname Linux 44f8f8cbf6c6 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24 17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 9b929f5
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~18.04-b07
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4347/3/testReport/
Max. process+thread count 2321 (vs. ulimit of 5500)
modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4347/3/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@saintstack
Copy link
Contributor Author

saintstack commented May 26, 2022

TestRollingUpgrade failed on each of the three test runs. Running it in a loop locally, it does not fail w/ the patch in place. Looking at the test failures, the complaint is from shutdown handler in QJM or hang waiting on JMX bean change, unrelated to this non-functional change inside the datanode.

@saintstack saintstack merged commit ae9d671 into apache:branch-3.3 May 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@saintstack @hadoop-yetus